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Abstract — Cocoa pods harvest is a process where 
peasant makes use of his experience to select the ripe 
fruit. During harvest, the color of the pods is a ripening 
indicator and is related to the quality of the cocoa bean. 
This paper proposes an algorithm capable of identifying 
ripe cocoa pods through the processing of images and 
artificial neural networks. The input image pass in a 
sequence of filters and morphological transformations to 
obtain the features of objects present in the image. From 
these features, the artificial neural network identifies ripe 
pods. The neural network is trained using the scaled 
conjugate gradient method. The proposed algorithm, 
developed in MATLAB ®, obtained a 91% of 
assertiveness in the identification of the pods. Features 
used to identify the pods were not affected by the capture 
distance of the image. The criterion for selecting pods can 
be modified to get similar samples with each other. For 
correct identification of the pods, it is necessary to take 
care of illumination and shadows in the images. In the 
same way, for accurate discrimination, the morphology of 
the pod was important. 

Keywords — Cocoa pod, Processing of digital images, 
Artificial Neural Networks. 

I. INTRODUCTION 

Cocoa (Theobroma cacao L.) has been cultivated in 
America for thousands of years. Currently, all types of 
cultivated cocoa are varieties of numerous natural crosses 
or human intervention; however, is possible to classify 
the cocoa into three broad groups: Criollo or native, 
Forastero or peasant, Trinitario or hybrid [1,2]. 

The Cocoa Pod is an indehiscent berry. The pods’ size 
oscillates from 10 to 42 cm in variable form (oblong, 
elliptical, oval, spherical and oblate), the surface of the 
fruit can be smooth or rough, the colors of the surface can 
be red or green in the immature state [3]. Some authors 
consider that the shape of the pod is a reference point for 
classifying cultivated cocoa trees. The classifying of 
cocoa by its shape divides among Amelonado, 
Calabacillo, Angoleta, and Cundeamor [1,4]. A 

www.iiaems.com 


significant determinant of pod ripening is the outward 
appearance. The ripening is visible as the colors of the 
pod's external walls change. Usually, the outer walls turn 
green or purple changing to shades of red, orange or 
yellow depending on the genotype [5]. 

The ideal condition for pods harvest is when they are ripe. 
It is essential not to let the pods overripening as they can 
contaminate with some fungal diseases. The state of 
overripening promotes the germination of seeds and 
causes quality defects. Green fruits should not be part of 
the harvest. The seeds of green pods are hard, they cannot 
be separated easily and do not ferment because the 
mucilage is not finished forming [6]. 

For a proper fermentation, it is recommended to separate 
the pods according to their shape, color, and size to avoid 
the combination of seeds varieties. Ripeness degree of the 
pods determines the quality of cocoa beans. No mixtures 
of pods with different ripen estate should be generated, 
mainly when it is sold to process fine chocolates. The 
main chocolate industries look for a unique origin and 
reject the mixtures of varieties [7]. 

In food, color is one of the factors considered by the 
customer at the moment of the purchase of a product. The 
color is a quality criterion in many classification 
standards. The color is a parameter used to evaluate the 
ripening status of fruits [8]. Thus, different technologies 
have been implemented to harvest fruits in their best 
conditions. 

Artificial vision systems incorporate color information. 
Within the range of visible wavelengths, some 
compounds in fruits absorb the light. These compounds 
are pigments, such as chlorophylls, carotenoids, and 
anthocyanins [9]. 

Tools commonly used in graphic design, such as digital 
cameras, computers and image processing software, can 
be used to process and analyze images. Image processing 
can identify the color of food s amples during fruit 
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ripening. The use of vision systems allows replacing the 
use of color charts, colorimeters and spectrophotometers 
[8]. Colorimeters has deficiencies to describe perceptual 
chromatic responses with a multitude of visual parameters 
[ 10 ]. 

Digital image processing involves methodologies for 
analyzing and finding quantitative information. The tools 
and algorithms for the capture of images allow evaluating 
in a non-destructive way [8]. Since a few years ago, these 
methodologies were applied. Some examples are listed 
below: Systems for recognition and localization of quasi- 
spherical objects by laser telemetry [11], processing of 
optical images of cherry coffee fruits by acoustic-optical 
filters [12], Systems to identify the color of the epicarp of 
tomatoes [13], a mobile platform for the collection of 
oranges [14], and analysis of fruits such as wheat, Feijoa, 
Mango, plantain and mandarins [15,16, 17,18]. 

On the other hand, the use of artificial neural networks 
(ANNs) and artificial vision has more application in the 
food industry recently. The ANNs prioritize the 
classification, the recognition of patterns and the 
prediction of crops [19]. ANNs try to emulate the 
behavior of the human brain. The ANNs extract 
knowledge from a set of data obtained during training. 
ANNs can be considered as models of calculation that 
uses very efficient algorithms, which operate in a 
massively parallel way. These artificial nets allow 
developing cognitive tasks like learning patterns, 
classification or optimization [20]. Most applications in 
ANNs are related to pattern recognition problems and 
make use of architectures such as multilayer perceptron 
[ 21 ]. 

The techniques of image processing and ANNs are 
applied as a methodology for classification of Royal Gala 
apples. In these techniques, a multilayered neural network 
was implemented through supervised learning and trained 
with different algorithms [22]. 

In this paper, we proposed the processing of images with 
artificial neuronal networks in the harvest of Cocoa pods 
using images taken by a photographic camera. 
Subsequently, from the objects contained in the image, it 
obtains a matrix of data. Finally, we process the data from 
the matrix using the data from previous training. This 
methodology improves the harvest process substituting 
the handcrafted way. With this methodology, the human 
perception of the color of the cocoa pod is removed and 
controls the quality of the harvested fruit. 

II. PROCESSING ALGORITHM 

The proposed processing algorithm is divided into four 
steps (Fig. 1). The algorithm processes an image in JPEG 
format with dimensions of 1920 x 1080 pixels using the 
RGB color space (Red, Green, Blue). The image should 
have good saturation and the least ambient noises. First 
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three steps remove the colors that are not typical of the 
ripe cocoa pods and gets the features of the found objects. 
An ANN evaluates these features. Therefore, the result of 
the algorithm is the identifier of the ripe cocoa pods on 
the input image. 



Fig. 1: Processing algorithm 


2.1 Color Enhancement 

The first step of the algorithm is to improve the colors of 
the input image. The lack of color intensity can be 
enhanced by adjusting the saturation values. For making 
it with less computational effort, it is proposed to convert 
the image in RGB color space to HSV (Hue, Saturation, 
Value). The three primary components that integrate the 
HSV color space are hue, saturation, and brightness [23]. 
The matrices of the HSV space has the same dimension as 
the RGB image. Assuming that values below 25% 
characterize obscure colors, the elements of the saturation 
matrix S are modified by, 

0.8, s Uj > 0.25 ... (1) 

S U , S U < 0.25 

Si.j c S’ 

S E Mmxn (C), 

C = {x E R , 0 < x < l), 

Rows i, 1 < i < m, 

Columns j, 1 < j < n. 

Consequently, the new saturation matrix holds the dark 
colors that could represent a flaw in the cocoa pods. To 
end this stage, the image in the HSV color space converts 
into the RGB color space. The conversion is done to 
avoid alterations in the properties of the colors during the 
following steps of the algorithm [23]. Figure 2 shows the 
performance of this stage of the algorithm. 
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(a) (b) 

Fig. 2: (a) Original image , (b) Enhanced image 


22 Segmentation 

Before the extraction of features, in the segmentation 
stage of processing, the image is prepared first. 
Environmental noises and unwanted objects are filtered in 
the image to acquire reliable features (Fig. 3). At the end 
of this stage, is get a binary image with smoothed edges. 

Segmentation 



i 


Morphological Transformation 


Fig. 3: Segmentation steps 
2.2.1 Color Filter 

In this segmentation stage, the RGB image is processed to 
remove several colors ranges through five filters. The 
colors excluded are white, blue, magenta, green, brown, 
and some red hues. Relational operators and logical gates 
form the color filters. The result of the filters is a binary 
image. 

The white filter creates a FI matrix that meets the 
conditions of Equation 2. The blue tonalities generate a 
second F2 matrix that complies with Equation 3. Magenta 
colors and red hues provide a third F 3 matrix that 
satisfies Equation 4. While the green color, is represented 
by the F4 matrix created by Equation 5. Finally, Equation 
6 represents the filter of yellow colors, brown, and some 
orange shades. The conditions of the last filter create the 


F 5 matrix. This final filter is calibrated according to the 
desired ripening of the cocoa pods. 


fhj 

= hj 

> 235) &(g u > 235) 

& (b t j > 235), • 

••(2) 

fhj 

= (hi 

^ 9i,i ) Hhj > r ij)' 


•••(3) 

fhj 

= h.j 

^ b i,j) & (9i,j - b jj f 


..(4) 

fhj 

~ C 9i,j 

> r i,j ) Hdu > hj )- 


"(5) 

fhj 

= ((la 

ij - h,j\) ^ y) & (0u 

> 9i,j)&(n,j > b 
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••(6) 

n,j c 

: R , 

9i,j c G , by C B , 



hu 

c FI 

, f2 u c F2 , /3;j 

c F3, 



fhj c F4 , f 5 Uj c F 5 , 

[R ,G , B ] G M mxn to), 

B = {xEN,0<x< 255}, 

[FI, F2 , F3 , F4 ,F5 ] G M mxn (K), 

K = { 0 , 1 }, 

y G N , 0 < y < 255, 

Rows i,l < i < m, 

Columns j, 1 < j < n, 

where R, G , and B represent the RGB matrix color space 
and, y represents the filter five setting to calibrate the 
color hue. 

The resulting matrix of filters is added to obtain a matrix 
containing all the excluded tonalities, 

FF = FI + F2 + F3 + F4 + F5. ••• (7) 

Therefore, the logical negation of FF provides the binary 
image with the required tonalities. Color filtering can be 
seen in Figure 4 

El = FT -(8) 


Fig. 4: Filtered and binarized image 
2.2.2Holes Filter 

The second segmentation stage removes the holes of the 
objects found in the binary image. It is proposed to 
eliminate these holes to improve the results in the 
morphology stage. Not all holes are covered. Those small 
holes resulting from reflections on the objects are filled 
in. Large holes generated by dark spots are not 
eliminated. Naturally the pods have small dark marks; 
however, large spots on a cocoa pod are a quality defect. 
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Identifying the holes consists of finding all the values of 0 
in the El matrix that are isolated from the values that 
make up the background of the binary image. The largest 
neighborhood group of elements in the El matrix with a 
value of 0 form the background of the image. Isolated 
elements are called holes. For identify holes, the "imfill" 
function of MATLAB ® is applied. The function 
produces a binary matrix H , which contains the holes 
founded in the input image and represents them with a 
value of 1 [24]. 

From if, an HI matrix containing holes formed by light 
reflections is obtained, 

H1=H&F1, •••(9) 

[H,H1] e M mxn (M). 

Also, using the methodology described in the following 
section (Area Filter) the H2 matrix is generated. This 
matrix represents the smaller holes. The matrices obtained 
are added, and a matrix is gotten with the holes to be 
filled, 


HF = HI + 

if 2 . 


•••( 10 ) 


Finally, the holes in the original image are filled through 
E2=E1+ HF, ••• (11) 

where E2 represents an image without holes. In Figure 5, 
the application of the Hole Filter stage can be seen in 
Figure 4. 



Fig. 5: Filtered image without holes 
2.2.3Area Filter 

There are small objects in the binary image gotten in the 
previous step. Small groups of pixels and isolated pixels 
with a value of 1, shape these objects. Removing these 
pixels from the background of the image makes hold large 
objects. The third stage of the segmentation, filter the 
objects by their areas. The methodology removes all 
neighborhood groups of elements with a value of 1 in the 
E 2 matrix For this, the areas of the neighborhood groups 
are analyzed in the matrix Using the "Blob Analysis" 
function of SIMULINK ® is possible to get areas [24]. 
From this function is obtained the data of the areas and a 
matrix of Bounding boxes. With the areas of the 
neighborhood groups found is created a vector column A. 


The matrix of BB contains the coordinates and size of 
Bounding Boxes. These enclose the neighborhood groups 
found in the input matrix entered into the function. 

In base on vector A, is created a reference vector Q , 

Rk,i = (a k> i < x), - (12) 

Rk, 1 Q > ^k, 1 A, 

A e M mxl (R + X 
Q e M mxl (ik), 

Rows k, 1 < k < z, 

where x is the minimum allowed area and z is the number 
of neighborhood groups found in the E2 matrix 
With the values of Q, a reference matrix WW is 
generated, which is created from the horizontal 
concatenation of Q with itself four times. The above is 
done to equal the dimensions of Q and BB. Based on 
WW the elements of BB are conditioned, 

,, _{bb kA , ww M = l -(13) 

kA l 0, ww k 4 — 0 

bb kA c BB , ww k 4 

c WW, 

BB e M mx4 (wd, 

W e M mx4 (M), 

Rows k,l < k < Z. 

Finally, over the elements of E2 is drawn with 0 
Bounding Boxes of BB. The matrix obtained from this 
procedure is called E 3 and characterizes the binary image 
filtered by area. This way, smaller objects are removed, 
and a cleaner image is obtained. Figure 6 shows the 
performance of this stage. 



Fig. 6: Image filtered by area 
2.2.4Morphological Transformation 
The last segmentation stage softens the edges of the 
objects in the output image. In some cases, the edges of 
the objects in the image show openings, and in others 
cases, they have outstanding reliefs. Modifying the edges 
in objects is known as morphological transformations. 
The basic morphological operations used here are known 
as dilation and erosion [25]. Morphological operations 
apply a structured disk element to objects. The disk 
structure is the best structuring element that smooth the 
edges in ovoid shapes. This structured element is 
subtracted or added to the edges and makes a smoother 
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perimeter. The modifications do not generate change in 
the dimension of the binary image. 

The first part of the transformation is to apply a dilatation 
followed by erosion to E3. This technique is known as a 
closing function, and its output is a new binary matrix. 
This function eliminates small apertures at the edges of 
objects. Then, an erosion is applied followed by a 
dilation. This last technique is known as opening and 
creates the resulting matrix E 4. With the opening 
function, is possible to remove small groups of pixels in 
the reliefs of the objects [25]. For the transformations 
mentioned above, the radius of the structuring element in 
the closing function must be less than the opening 
function. In Figure 7 the changes are appreciated 
comparing with Figure 6. 



Fig. 7: Image with morphological transformation 

2.3 Features Extraction 

This stage gets the features of the object in the image. 
These features are used to discriminate objects, which do 
not have the shapes of cocoa pods. Using "Blob Analysis" 
from SIMUUNK ® are gotten these features and the 
matrix of Bounding Boxes [24]. The features obtained for 
this processing were: major axis, minor axis, eccentricity, 
and extent. The extent feature represents a relationship 
between the area of the found object and the area of its 
Bounding Box [26]. Features extracted are in the form of 
column vectors. Features of the major axis and minor axis 
are joined to represent the following relation: 


0 


i,i 


U = 


^1,1 

@ 2,1 


V 2,l 


... (14) 


- v k,i- 

°k, i c O , v K1 c V, 

v e >f mxl ( R + ), 

OGM mxl (R + ), 

U e Mmxl ( C), 


Rows k, 1 < k < z lf 


where U represent the result of dividing the axis vectors, 
0 the minor axis, V major axis, and z 1 the number of 
objects in the E 4 image. 

There are two significant advantages of the features 
obtained at this stage. The first is the performance of the 
three features (relation of axes, eccentricity, and extent) 
over ovoid objects. Finally, the value of the features tends 
to be retained with the distance; as shown in Figure 8. 


t 

Minor Axis 

Eccentricity - Extent 

Major Axis 


0.784 0.620 0.767 

(a) 


Minor Axis 

Eccentricity - Extent 

Major Axis 


[ 0.782 0.621 0.776 | 

(b) 

Fig. 8: (a) Pod about a meter away, (b) Pod two meters 
away. 

Finally, in this stage is generated the matrix of BF 
Bounding Boxes that enclose the found objects and the 
matrix C that contains the features of the objects. Matrix 
C come from the horizontal concatenation of the three 
vectors containing the features separately. 

2.4 Artificial Neuronal Network (ANN) 

In this last stage, the ANN received the features gotten in 
the previous step. Each feature creates inputs to the 
Artificial Neuronal Network and, the output relates the 
inputs by functions [27]. Therefore, the network will be 
made up of three inputs and a single output. The output is 
a Real number between 0 and 1. The approximation of the 
output to 1 indicates that the values of the features come 
from a cocoa pod. 

Within the three fundamental operations of the artificial 
neuron, the transfer function / is related to the type of 
output (Fig. 9). From the existing transfer functions, the 
sigmoidal function is selected for this application. This 
function takes inputs between negative infinity and 
positive infinity. The output of the sigmoidal function 
generate values between Oand 1 [27]. 
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Fig. 9: Single input neuron [27] 

The output of a simple artificial neuron with sigmoidal 
transfer function is described by, 
t= logsig (wp + h), ••• (15) 

[w,p,b] E R, 

tec, 

where t is the network output, p is the input variable to 
the neuron, w is the net weight and b is the bias. 

A single-layer ANN is deficient for this processing 
algorithm. Therefore, a multi-layered ANN has been 
designed. The type of ANN proposed here will be a 
Feedforward Neuronal Network (FNN). This type of 
networks behaves with good results in pattern recognition 
and object classification applications. The number of 
layers used in this work is two, as recommended by [27]. 
Finally, the designed network has nine neurons in its first 
layer and a single neuron in the output layer (Fig. 10). 

Neural Network 


Hidden Output 



9 1 


Fig. 10: Feedforward Neuronal Network 

Feedforward Neuronal Network can use the scaled 
conjugate gradient method for training. For large 
networks, this training algorithm is the most efficient. The 
pattern recognition problems frequently apply this method 
[27]. 

The methodology of the FNN designed is to read the data 
of C. Each row contains the features of an individual 
object. The single output of the network will be based 
mainly on the features extracted and weights of the FNN. 
The result of the network will be a scalar between 0 and 
1. This value is conditioned by, 
t ± = t >1 , ••• (16) 

l e C, 
t ± e 

where t is the output of the FNN, t ± is the conditioned 
output and l is the criterion that conditions the output. 
This criterion is in function of the percentage of the 
output coincidence with the samples in the training. 


With the values of t 1 is created the reference column 
vector 01. This vector comes from the vertical 
concatenation of the outputs of the FNN. 

Based on 01, the WF matrix is generated, whose values 
are gotten from the horizontal concatenation of Q1 with 
itself four times. The above is done to equal the 
dimensions of Q1 and BF. Then, the elements of BF are 
conditioned in the following way, 



' J f\, 

bfk ,4 c BF, wf kA c WF, 

BF £ M mx4 ( R), 

W E M mx4 (M), 

Rows k, 1 < k < z ± . 

Finally, the image processing algorithm takes the BF data 
to draw the Bounding Boxes that point out to the cocoa 
pods on the RGB input image that this algorithm 
received. Using the SIMULENK ® "Draw Shapes" block, 
draw the Bounding Boxes over the RGB image. 

III. RESULTS AND DISCUSSION 

The methodology proposed in this research was 
developed in MATLAB ® R2018A using the Visual 
Programming Environment SIMULENK ®. The 
simulation was done on a computer with Microsoft ® 
Windows ® 7 Home Premium, Intel ® Core ™ i7- 
2630QM 2.00 GHz processor and 8 Gb DDR3 1333Mhz 
RAM. 

The images used for the development of the investigation 
were obtained from cacao farms of the village 
Cucuyulapa. This locality is part of the municipality of 
Cunduacan, Tabasco, Mexico. In the locality, images 
were taken with the natural daylight of midday. All the 
images taken had the same illumination and focal 
aperture. The images capture was taken at distances 
between 1 and 2 meters. 

The types of pods used in this experiment were hybrids of 
criollo cocoa with an amelonada shape. The observed 
color changes in these pods were from green to yellow. 
Some variations in the length of the pod were observed in 
the same cocoa genotype. 

An FNN of nine neurons in their hidden layer and a single 
neuron in the output layer process the images. The 
training following an algorithm that trains, validates and 
tests the results. The training algorithm uses, 175 images 
of cocoa pods, of which 70% were used for training, 15% 
for validation and another 15% for testing. Training 
algorithm got a gradient of 0.294 final at 39 iterations. 
The results obtained are shown in Table 1. Consequently, 
minimizing cross-entropy in training results in accurate 
pod identification, while decreased percent error reduces 
mistakes relate with pods and erroneously identified 
objects. 
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Table 1: Training results 



Cross-Entropy 

Percent Error 

Training 

5.77386e-l 

6.50406e-0 

Validation 

9.35257e-l 

0 

Testing 

9.95469e-l 

0 


The identification of the objects leads us to create a 
confusion matrix. This matrix has the erroneous or 
successful percentages of the FNN to obtain its targets. 
Figure 11 represents the total confusion matrix of the 
training algorithm processes. The percentage of total 
assertiveness towards to the wanted Targets is 95.4%. 


All Confusion Matrix 


49 

3 

94.2% 

28.0% 

1.7% 

5.8% 

5 

118 

95.9% 

2.9% 

67.4% 

4.1% 

90.7% 

97.5% 

95.4% 

9.3% 

2.5% 

4.6% 


o \ 

Target Class 

Fig. 11: All confusion matrix from FNN 

The processing algorithm was tested using 23 images of 
cocoa pods different from those used in the training 
algorithm. The tests are in charge of evaluating the 
performance of the algorithm for each image at different 
values of the l match criterion (Equation 16). Value of l 
evaluates the image more stringent. A high value in l 
allows making a homogeneous selection of pods in good 
shape and with proper ripening. According to the ripening 
of the pods, Table 2 shows the most appropriate values of 
l for the algorithm. 

Table 2: Test results 


i 

Percent Assertiveness 

0.7 

89% 

0.8 

91% 

0.9 

91% 

0.95 

83% 


The values of l between 0.8 and 0.9 gets the best 
performance of the algorithm. Criterion of 0.95 presents a 
lower percentage of assertiveness because it penalizes the 
shape of the pod. The criterion of 0.95 is suitable if it is 
looking for pods that have the proper ripening and the 
average shape that characterizes the selected cocoa 
genotype. Finally, the criterion of 0.7 shows good results 


to select ripe cocoa pods, but with the disadvantage of 
confusing some pods with some green hue. Then, in 
Figure 12, the result of the algorithm can be seen using 
0.9 as the criterion for evaluating the sample. 



(a) (b) 

Fig. 12: (a) Image inputted, (b) Image result 


The errors generated in the tests came from the values of 
the features in the pods. These values are affected by the 
length of long pods and pods with green color or spots at 
its ends. At the time of evaluating the features of these 
types of pods with considerable deviations in their form, 
the algorithm can recognize the samples as 
morphologically correct pods. 

IV. CONCLUSION 

In this work, a methodology was proposed to identify ripe 
criollo cocoa pods type amelonado. It could be observed 
that these fruits tend to change from green to yellow. The 
overripening make color changes on the pod to yellow to 
orange hues. Therefore, it was proposed to use a filter in 
which the desired tonality of the fruit could be adjusted. 
Also, the measurements of the features of the pods were 
essential to identifying if the pod presents damage for 
some disease. 

The images taken for this algorithm were carried out 
taking care of lighting and light reflections. These factors 
cause noises that impair the calibration of the camera's 
colors and the adjustment of the desired hue of a pod. The 
modification of the saturation of the image helped to 
improve noises that derive from these factors; however, 
excessive shadows and reflections cause problems in 
digital image processing. 

The use of the area filter helps the shapes in the images 
do not have so many morphological changes and to be 
able to eliminate objects that cause noises in the binary 
image. The previous methodology proposes to maintain 
as much as possible the original size of the forms and 
despite this eliminate noises by a group of pixels. 

Features used to evaluate the objects in the images, gave 
optimal results to discriminate between pods and different 
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shapes. It could be seen that the features retained their 
value, regardless of the distance of the photograph. The 
performance of features represents an advantage, allowing 
selecting similar shapes without the need for a reference 
distance. 

The technique of modifying the criterion of selection of 
samples in the processing algorithm behaved in the 
manner expected. In this way, it is possible to adjust the 
criterion making it more critical and having homogeneous 
samples. Getting good results within 80% and 90% of 
coincidence criterion with training samples. If the study 
objects have many variations in their shape, setting the 
selection criterion may not be the most appropriate. 

It is expected in future work to study the variations of 
pods of different genotypes. This study could select ripe 
cocoa pods from all types of shapes. The selection of 
these pods could be made without being confused with 
other objects. In the same way, the features should be 
improved or complemented, to avoid confusion between 
ripe pods and pods with possible morphological diseases. 
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